Conventionally, an in-vehicle camera comes into use for the purpose of supporting a driver to confirm the surroundings. In addition, in recent years, a system is put into practical use in which videos captured by a plurality of cameras attached around the vehicle are transformed in viewpoint, and an overview video (a video viewed from right overhead) around the vehicle is generated by synthesizing the videos and presented to a driver.
As described above, in order to generate a synthesized video by transforming the videos captured by the cameras, camera-specific information (inner parameter) such as an optical characteristic (a focal distance and a lens distortion) and a size of an image pickup device, and information (external parameter) on an attached position and an angle of each camera are required. The videos captured by the cameras are transformed in overview using camera parameters obtained by synthesizing the inner parameter and the external parameter as described above. A video captured from the overview point can be virtually generated by synthesizing the overview videos obtained from the videos.
By the way, the cameras are attached to the vehicle at positions and with angles in conformity to design values. At that time, there occurs an error inevitably. In a case where the overview video is generated on the basis of the design values regardless of such an error, the overview video captured from the overview point is necessarily not an expected one, and a deviation occurs in the video by an amount of the error. In particular, in a case where a plurality of videos are synthesized, an influence of the deviation remarkably appears in a boundary area of images of the cameras in the synthesized image, which is greater on appearance than a case where a single camera is used. In order to solve such a problem, a correction (called calibration) of the error caused from the design value of the camera parameter is performed.
Since the error is necessarily corrected with an extremely high accuracy in the calibration, a method of estimating a current installation state of the camera from the captured video is employed in place of a method of mechanically adjusting the installation state thereof. As a typical method of estimating the installation state from the captured video, there is typically employed a method of accurately providing a pattern (a calibration chart) printed in a sheet or a plate at a predetermined position, and correcting the camera parameter such that the actually captured video is matched to a video captured by a camera which is manufactured and provided in conformity to the design value. At that time, the attachment state of the camera is not adjusted, but numerical values in a program related to the attachment position and the angle of the camera are corrected through an image transformation.
By the way, when the calibration is performed in a production line of a factory at the time of vehicle shipment, the calibration is executed by simulating an empty condition of no one passenger or a specific loading state such as a case where a driver sits in a driver seat. Therefore, the deviation is not generated in the video in the same state as the actual calibration such as the empty state or the specific loading state.
However, when the user actually uses the vehicle, the states such as the number of riding persons, a seating place, and a loading state of a baggage are variously changed. Then, when the loading state of the vehicle is changed, the posture of the vehicle is also changed, and accordingly the installation state of the camera with respect to the ground surface is also changed. In other words, an error is generated since the camera parameter varies. Therefore, the deviation is generated in the video by the error of the camera parameter.
With regard to such a problem, PTLs 1 and 2 disclose technologies of correcting the camera parameter in running of the vehicle.
An online calibration method of a vehicle camera disclosed in PTL 1 is a method in which an adjacent portion of a road is captured by at least two cameras, a road characteristic in a longitudinal direction is specified in an image frame, a road characteristic in the longitudinal direction specified in at least two image frames captured by two camera is selected such that two image frames are matched by a single line therebetween, a matching rate of the single line is analyzed to determine an offset of the line between two image frames, and the offset of the determined line is applied for the calibration of the camera.
In addition, an online calibration method of a vehicle camera disclosed in PTL 2 is a method in which a part of a road is captured by the camera, a road characteristic in the longitudinal direction is specified in an image frame, a point along the specified road characteristic in the longitudinal direction is extracted and the extracted point is transformed into a virtual road plane by a perspective mapping in consideration of a given camera parameter, the extracted point thus transformed is analyzed with respect to the vehicle to determine a deviation from a line in parallel to the vehicle of the point, and the measured deviation is used to define an offset correction of the camera.